Skip to content

Dynamically retrieve the cluster audiences#49796

Merged
hugoShaka merged 6 commits intomasterfrom
hugo/fix-kube-default-audience
Dec 5, 2024
Merged

Dynamically retrieve the cluster audiences#49796
hugoShaka merged 6 commits intomasterfrom
hugo/fix-kube-default-audience

Conversation

@hugoShaka
Copy link
Copy Markdown
Contributor

@hugoShaka hugoShaka commented Dec 4, 2024

Fixes #49756

I introduced a bug in #49528 by using a standard hardcoded audience for in-cluster joining. This PR makes Teleport detect the audience by looking at its own token.

Not every cluster is using kubernetes.default.svc as the default audience, some are using kubernetes.default.svc.cluster.local or custom values.

Changelog: Fix a bug breaking in-cluster joining on some Kubernetes clusters.

Comment thread lib/kubernetestoken/token_validator.go Outdated
@hugoShaka hugoShaka force-pushed the hugo/fix-kube-default-audience branch from 9a28938 to 6443509 Compare December 4, 2024 23:38
@hugoShaka
Copy link
Copy Markdown
Contributor Author

hugoShaka commented Dec 4, 2024

Cut a v18 test build and this fixes the issue:

helm repo add teleport-staging https://charts.releases.development.teleport.dev
helm repo update
helm upgrade --install --namespace teleport-cluster --create-namespace teleport teleport-staging/teleport-cluster --version 18.0.0-dev.hugoooops.2 --set clusterName=hugo
kubectl get pods -n teleport-cluster

NAME                              READY   STATUS    RESTARTS   AGE
teleport-auth-7c77b85b9-7gjv8     1/1     Running   0          15m
teleport-proxy-84c9b8586c-t7kr8   1/1     Running   0          15m

Comment thread lib/kube/token/validator.go Outdated
Comment thread lib/kube/token/validator.go Outdated
Comment thread lib/kube/token/validator.go Outdated
return nil, trace.Wrap(err, "reviewing token and retrieving audience")
}

return reviewResult.Status.Audiences, nil
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The API docs for status.audiences mention

a TokenReview returns an empty status.audience field where status.authenticated is "true", the token is valid against the audience of the Kubernetes API server.

Can we be reasonably sure that all supported kube api servers will actually populate the field?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Kubernetes doesn't guarantee that the audiences field is always defined, we should prioritize parsing the JWT token directly, as this ensures we know the field is set. My concern is that, since the documentation doesn't mandate it, there might have been a version in the past that didn't include the audiences field, or a future version might omit it. To safeguard against this, we should handle the token by unmarshaling it ourselves.

You can use jwt/v4 package for that

import "github.com/golang-jwt/jwt/v4"
...
p := jwt.NewParser()
	claims := &jwt.RegisteredClaims{}
	_, _, err := p.ParseUnverified(token, claims)
	if err != nil {
		panic(err.Error())
	}

	fmt.Println(claims.Audiences)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(there is a (go-jose/v3/*jwt.JSONWebToken).UnsafeClaimsWithoutVerification method fyi)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed to parsing our own token without validating. The missing audience thing is for legacy Kubernetes tokens (which are in theory still supported, but we do reject them for kube >= 1.21).

So basically:

  • if there's an audience on our token we know this is a modern kube and can set the audience
  • if there are no audience on our token, this is an old kube and we do the review without specifying audience, as we were doing before

Comment thread lib/kube/token/validator.go Outdated
Copy link
Copy Markdown
Contributor

@bl-nero bl-nero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After fixing issues pointed out by Edoardo, this is good to go.

@hugoShaka hugoShaka requested a review from espadolini December 5, 2024 16:05
Comment thread lib/kube/token/validator.go
Comment thread lib/kube/token/validator.go Outdated
Comment thread lib/kube/token/validator.go Outdated
Comment thread lib/kube/token/validator.go Outdated
Comment thread lib/kube/token/validator.go Outdated
@hugoShaka hugoShaka enabled auto-merge December 5, 2024 18:34
@hugoShaka hugoShaka added this pull request to the merge queue Dec 5, 2024
Merged via the queue into master with commit d9c4906 Dec 5, 2024
@hugoShaka hugoShaka deleted the hugo/fix-kube-default-audience branch December 5, 2024 19:26
@public-teleport-github-review-bot
Copy link
Copy Markdown

@hugoShaka See the table below for backport results.

Branch Result
branch/v15 Create PR
branch/v16 Create PR
branch/v17 Create PR

carloscastrojumo pushed a commit to carloscastrojumo/teleport that referenced this pull request Feb 19, 2025
* Dynamically retrieve the cluster audiences

* hard fail

* remove unused variable

* Address edoardo's feedback + ensure support of older kube versions

* Address edoardo and tiago's feedback + dedup audiences

* lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[helm/teleport-cluster] Add audience into serviceAccountToken for proxy service

4 participants